feat(postgres): add destination with automatic schema evolution - #926
feat(postgres): add destination with automatic schema evolution#926riderx wants to merge 3 commits into
Conversation
Add a Postgres destination that auto-creates schemas/tables and applies supported DDL (add/drop/rename columns, defaults, DROP NOT NULL) during replication so PG→PG pipelines stay current without manual migrations. Co-authored-by: Cursor <cursoragent@cursor.com>
282d36f to
c353068
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c353068a5e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Hi, thanks for this PR! We are not accepting new destinations at this moment due to the high workload that the team has, but we are evaluating a Postgres destination in the future. May I ask for which use case you added this? Did you already try it out in staging/production? |
Encode source schema into destination table names when destination_schema is set, and add API primary-key validation so Postgres publications fail early instead of at copy time. Co-authored-by: Cursor <cursoragent@cursor.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
@iambriccardo thanks for taking a look, and totally understand the bandwidth constraint. I'm a paid Supabase Team customer. Today I run custom Postgres replication into Google Cloud because Supabase Pipelines / managed replication doesn't cover my needs:
So I currently use basic Postgres logical replication to those destinations. That works for row data, but schema sync is missing — DDL on the source (add/drop/rename columns, etc.) does not propagate cleanly, which is the gap this PR targets: a Postgres destination with automatic schema creation + evolution on top of ETL's existing CDC/schema pipeline. I've exercised it locally with the destination integration tests in this PR (initial copy, CDC upsert/delete/truncate, schema add/drop/rename, and copy reset). Not in production yet — happy to keep iterating if/when a Postgres destination is back on the roadmap. Also addressed the Codex review notes in the latest commit: schema-override table-name collisions and API primary-key validation. |
Keep the Postgres destination feature while adopting main's ClickHouse schema-rewind docs and simplified egress feature wiring. Co-authored-by: Cursor <cursoragent@cursor.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
2 participants
Summary
DROP NOT NULL); reject partial/UnchangedToast updates and handle PK-changing updates via delete-then-upsert.DestinationConfig/ replicator / API support, docs, and integration tests for copy, CDC, truncate, schema evolution, and copy reset.Test plan
cargo nextest run -p etl-destinations --features postgres,test-utils -- postgres(8 passed)cargo check -p etl-destinations --features postgrescargo check -p etl-replicator --features postgrescargo check -p etl-apiMade with Cursor